Repository navigation
GLM-5.3-Flash on two CUDA ranks, in families/glm - #855
BobClawblaw wants to merge 45 commits into
Conversation
….cu) and the ranks' control link (tp_link.zig): small gathers as RDMA writes over both halves of the port, after b12x's exchange; NCCL carries the rest
…wer: batched prompt passes (prefill_many, addStreams), a backend's refusals for a memory guard, follow() for a tensor-parallel rank 1, shared-pool admission fields, image and video inputs for the native server (vision_inputs, media_fetch, qwen_image, stb_image)
…glm_cuda), its kernels (zig/kernels/cuda/glm: the dense matmul at 4/6/8/16 bits, the EXL3 trellis experts with the fused prompt kernel, latent attention, the indexer, KDA, the hyper-connections, the vision tower's helpers, DFlash2), tf-cuda-test glm-batch/glm-prof/glm-tf, -Dglm (cherry picked from commit 611959f15964cda97ecd12d0e459ea4ac0e84f78)
…h), no include across families
…prefill, sample) and no Flash Next test commands
…meration, kernel parameter info, virtual memory)
|
@BobClawblaw thanks for the CUDA GLM work. Here are test results for Setup. 2× DGX Spark (GB10), one direct ConnectX-7 cable, with two HCAs configured in Checkpoint: Engine test.
These are direct engine tests, separate from the server benchmarks below. Both ranks printed the pass line, but rank 0 was still alive about 22 minutes later, with Server tests.
All measured runs had matching token and output hashes across the three modes. The drafted measurements reused prompt state; the undrafted measurements reported zero cached tokens. With DFlash2 and Median aggregate decode throughput at level 4 was 97.4 tok/s for These are TensorFold's benchmark-tool prompts, so the rates are not directly comparable with the recipe's prose/code/structured/list results. DFlash2's observed throughput was 6.3% higher than MTP on Prompt TTFT. MTP,
The server counted 2,040–32,762 prompt tokens. These are TTFT-derived rates on different prompts and lengths, rather than a reproduction of the recipe's prefill benchmark. Rank 0's startup report labelled 85.65 GiB as model allocation with DFlash2 loaded. Its Not tested: |
|
Thank you for the independent run, that is exactly the check this PR needed. The decode gap against the recipe's numbers is settings, not hardware: the recipe serves with Rank 0 alive after "glm-batch passed": known and not yet fixed in this branch; our harness kills the test binaries on the pass line, which is no answer. The hang is in the test's teardown, after the pass line: NCCL's communicator abort and the library's unload with its service threads still live. The fix lands as a commit here shortly: each "no prompt cache" at startup while responses report cached tokens: the startup line describes the lane core's prompt cache, which this family does not use; the family keeps its own prompt states (what the The 1M-context configuration is measured in the recipe's evidence ( |
|
Both fixes are on the branch as 8f99433: |
…ardown hung in NCCL's service threads under the library's unload, reported on PR 855); the startup line says a family keeps its own prompt states instead of 'no prompt cache'
3c6ed00 to
87ef757
Compare
…cel; lean rule and zig fmt setup() returned cancel()'s true for a background warm pass that yields to a newer request, so the caller launched the job cancel() had just finished (SEGV in lane_host_warm_test). It now returns false, as for a drop. Comments one line each; lean_check skips zig/vendor/ (stb_image.h, as on ashhart#726). lane_host.zig and lanes/engine.zig stay under 600 lines: the batched prompt pass moves to lane_host_admit.zig (many, one) and lanes/open.zig (addStreams, adoptPrefilled), aliased where they were.
…ders in docs/glm-cuda-notes.md Splits move code unchanged (only imports, aliases and pub change): kern.zig into kern_attn, kern_kda and kern_moe; weights.zig into ckpt, experts_load and qpack; dflash.zig into dflash_load and dflash_select; native.zig into native_draft, native_prompt, native_follow and native_env; exl3.cu's transposed prompt design into exl3_transposed.cuh; tf-cuda-test glm-tf and glm-prof into glm_forward_run.zig.
…cs/roce-gather.md, zig fmt
…rozen words (the errors/content-image golden)
|
CI fix. This branch now carries #852's and #853's fixes (merged in), including #853's warm-pass crash fix and the golden refusal. Its own changes, in
Locally: |
…nsorfold-native takes memcpy from libSystem again (zig build test on macOS)
|
Hello @BobClawblaw, Thank you for GLM-5.3-Flash on two CUDA ranks. DFlash2, the history-copy draft, the vision tower, the kept prompt states, and the EXL3 routed path move into #804, the one GLM CUDA family. The two-rank link and the small RoCE gather move into the one transport. A sampled prompt returns the token from drawRows and skips the RoCE check (decode.zig:222, decode.zig:90). readPicks calls it (decode.zig:31). A timed-out gather writes nothing and returns success (roce_gather.cu:99), so the token can be the previous request's candidates. Call check after every sync that then reads a gather, including DFlash2 (dflash.zig:626). fits still uses this rank's unrounded max_bytes (roce.zig:453). The handshake compared only the rounded slots (roce.zig:213). Startup agrees only that the drafter is on both ranks (native.zig:381). Each rank reads its own prompt chunk (forward.zig:31, decode.zig:174), TF_GLM_MIX (native.zig:337, native.zig:781), and the DFlash2 split (dflash.zig:321, dflash.zig:718). Agree those before the first collective, or refuse the pair. The image frame goes out before the prompt is admitted (native.zig:576, native.zig:517). Rank 1 keeps the features and attaches them to the next prefill (native.zig:952). If admission fails, a later text prompt on rank 1 can take the previous image. Bind the features to that prefill, and drop them if it never arrives. release sends keep before the retained prefix exists (native.zig:913). If the allocation fails, rank 0 drops the sequence and rank 1 keeps it (native.zig:1003). Allocate first, or send an eviction when the keep fails. Thank you again. |
…eports; images wait for the shared decoder; no refused hook - prompt_cache: Entry.pins, Store.pin/unpin. victimBut passes pinned entries over, so a later job's lookup in admit.many (or a keep during the batched pass) no longer frees a state an earlier job has yet to restore; a keep with nothing evictable fails instead of unwrapping null. LaneHost.setup pins, reported and requeue unpin. - vision: zig/vendor/stb dropped and no longer compiled. qwen_image.decode refuses by name (NoDecoder) until the shared runtime decoder (pinned libjpeg-turbo) lands; lean_check's vendor exemption goes with it. - lanes: Backend.VTable.refused, Engine.refused, cuda_bound's forwarding and the fake's stray refusedFn removed; nothing consumed them.
|
@SvangenStudios thanks for rerunning On the 4–7% gap, I reran
Single runs drop by 7–17% at random on either commit, more than the gap between your two runs. The medians agree within noise, so I don't see a regression. @ashhart thanks for closing the four findings. Now that 1.0.12 and #804 are on main, I'm moving this work where you asked: the RoCE gather and the ranks' link into the transport on main (
I'll update this PR when the ported head passes them on our pair, and rerun |
# Conflicts: # zig/src/cuda/root.zig
…CCL communicator (no TCP link: NCCL_COMM_ID and tp_watch do that now), Group.openRoce brings it up on two ranks, allGather takes it by size alone, Group.check after a sync that reads a gather; tf-cuda-test roce-gather (every size against NCCL's bytes, graph replays, time a call)
…h (core/picture, server images and videos, media_fetch), its two-rank serving (--tp/--rank, engines.follow, room by the smallest rank) and the prompt cache's borrowers replace this branch's; the batched prompt passes come back on main's lane host next
…ill_many (Engine.addStreams; the lane host takes, sets up and launches each job as before and prefills the shared-pass ones together); a pass that errs or has no room leaves none prefilled and each stream goes alone; the CUDA and recorder wrappers forward it; the fake's batched backend and four host tests
…) into pr/glm-flash-cuda: the glm_cuda family leaves; DFlash2, history-copy drafts, the vision tower and the EXL3 routed experts move into families/glm commit by commit, its kept states are main's
…he main stream's all-gathers (each layer's partials, the MTP head's, the greedy pick padded to 16 bytes) take its RoCE writes, side-stream prompt exchanges stay on NCCL; Group.check after every draw that read gathered rows
…lis experts, the rest bf16): admitted beside ModelOpt NVFP4, loaded by the rank's 128-column cut, run by glm_exl3 (grouped decode, fused prompt) on an 8-slot plan into the MoE block's fp32 slot rows; bf16 dense layers through b16; the SwiGLU limits keep a NaN; tf-cuda-test glm-exl3 against float64 from core's EXL3 decoder
…/121 usage (tf-cuda-test kernel-contract-limits)
|
@ashhart a question before I bring this onto 1.0.12, since #804 landed GLM-5.3-Flash on CUDA from the NVFP4 checkpoint. The two families overlap:
I'll hold this branch until you say. |
… dies stops the others instead of leaving them waiting, a failed follower leaves with _exit; every peer's send/recv and NCCL's rings connect before the weights load, so a full GB10 never registers them (ibv_reg_mr refused the first prompt pass's split exchange)
…ore's copy proposals reach 7 tokens; head chains stay 3 deep and a first copy as wide as a head round (EXL3 on two GB10s: a rewrite-this-code reply 34.2 to 41.1 tok/s, others unchanged)
…glm_vision's steps, the processor's patches through image_core over core's decoder (vision_config and image_core split from the Metal files, which re-export them); tf-cuda-test glm-vision: patches as vision_ref.py's within 4e-7, features 1.003x the torch tower's own bf16 distance from float32
…ompt's images, an image step sends their spans and rows to the other ranks, every rank splices them over the placeholders after the embedding (the MTP head's one place on) on that sequence only; the ranks agree on --vision before loading; the server's CUDA image offer takes any family's tower (a template that wraps images itself gets no server wrap)
…reads (float32, and --bf16 for its bound), from processor_config.json
…: weights tiled in place at load, decode windows in K slices with dependent launches, prompts in one launch of 128-row blocks sharing weight tiles in L2 (TF_GLM_MM=0 keeps the glue's b16); TF_GLM_PROF=1 times a verify's sections; tf-cuda-test glm-mm-bench (EXL3 on two GB10s: a 2-row round 67.4 to 63.3 ms, one stream 4.5-9% faster)
…rward, draw, the MTP head's pass, commit), by size
…ilies/glm on main's transport; tree unchanged
BobClawblaw
changed the title
|
@ashhart the port is up: this branch now builds on What didn't come along yet, each as its own follow-up PR: DFlash2, 16 streams in one step (the family shares windows across 8 for now), the old family's draft rule and any-position prompt resume, and video (needs |
|
Thank you, @BobClawblaw, moving the port onto families/glm and tp_group follows the single-family direction and keeps your new image checks useful. We will land the transport through #852, then carry your GLM increment onto #804 with the selected shared EXL3 module. Your owned tower, sequence-bound images, state/watch work and fixed-MCG4 format checks come across with credit. We will retain released NVFP4/default arithmetic and final rank/image/token gates, then take video in #976. DFlash2, 16 streams and arbitrary-position resume remain the separate follow-ups you named. The family/module choices are settled; we can handle the shared wiring and failure-owner fixes at landing. Posted by Ash's AI Assistant Synthena |
3 participants
GLM-5.3-Flash on two CUDA ranks, in
families/glmAs asked on this PR, the two-rank GLM-5.3-Flash work now sits on main's family (
zig/src/families/glm) and transport (tp_group) instead of its ownglm_cudafamily. The branch is main + #852 (RoCE gather intp_group) + #853 (lane core: several queued prompts in one pass), then:tp_group.Group. On two ranks, the main stream's all-gathers (each layer's partials, the MTP head's, the greedy pick) use its RoCE writes; side-stream prompt exchanges stay on NCCL;Group.checkafter every draw that read gathered rows.glm_exl3runs grouped decode and the fused prompt kernel. Kernel-contract records for its eight launches.tf-cuda-test glm-exl3checks it against float64 from core's EXL3 decoder.tp_watch): a rank that fails or dies stops the others instead of leaving them waiting. Every peer's send/recv and NCCL's rings connect before the weights load, so a full GB10 never has to register them late.--vision): rank 0's tower (cuBLASLt plusglm_vision) encodes a prompt's images, and an image step sends their spans and rows to the other ranks, which splice them over the placeholders.vision_config.zigandimage_core.zigare split out of the Metal files (which re-export them), and patches go through core's decoder. The server's CUDA image offer takes any family's tower. The ranks agree on--visionbefore loading.tf-cuda-test glm-visioncompares againsttools/glm_vision_ref.py: patches within 4e-7, features 1.003x the torch tower's own bf16 distance from float32.glm_mm: weights tiled in place at load, decode windows in K slices with dependent launches, prompts in one launch of 128-row blocks.TF_GLM_MM=0keeps the glue's b16.TF_GLM_PROF=1times a verify's sections and prompt chunks.tf-cuda-test glm-mm-bench.Checks
CI's no-GPU steps pass on the head:
zig build test,test-kernel-contract,test-golden,fuzz-cortheon,test-cuda-host,zig fmt --check, lean check (0 new). The branch merges cleanly with current main.Two-rank checks on two DGX Sparks (GB10, TP=2), including the two you asked for:
Quality (EXL3 checkpoint, greedy) matches the
glm_cudafamily's published suite:glm_cudaNot in this PR (follow-ups)
glm_cudafamily's published speeds were measured with DFlash2, which is why its one-stream numbers are higher than this head's. DFlash2 onfamilies/glm: GLM-5.3-Flash on two CUDA ranks: DFlash2 drafts (--drafter) #979.glm_cuda's copy rule accepted more drafts at long context, and it resumed a kept prompt state at any position (main keeps them on the 2048-token grid).libtfvideo.prefill_many(lane core: several queued prompts in one pass (prefill_many, addStreams, the lane host admitting together) #853) for this family.